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ABSTRACT 

This paper describes t^o methodologies designed to 
address deficiencies in training literature, i.e., procedures which 
empirically evaluate the content validity of training programs, and 
strategies linking training programs to needs reassessment and 
program redesign. In section 1, a methodology for establishing the 
job relatedness of a police recruit training program is presented. A 
description of the program is given and the design issues faced by 
the researchers are described in detail (strategies for linking 
content validity with a modified job element approach to job 
analysis) including identification of content domains^ and evaluation 
of job relatedness. In section 2, a methodology for establishing the 
critical link between training evaluation and training program 
reassessment and redesign is presented. The technique, matching 
training emphasis and training needs, is conceptualized and presented 
graphically in a series of figures. A reference list and a series of 
nine tables illustrating the steps discussed in the two methodologies 
are also included. (PAS) 
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As noted by Goldstein (1980) in his review of the training literature, no 
procedure currently exists which empirically evaluates the content validity of 
an established training program. Unfortunately, the existing training liter- 
ature also fails to provide strategies which link training evaluation with 
training needs reassessment and program redesign. The goal of this presenta- 
tion is to describe methodologies which address both deficiencies in the 
literature* First we present a methodology which links a job element approach 
to job analysis with Lawshe's (1975) content validity ratio approach to esta- 
blish the job relatedness of a police recruit training program. Next we 
discuss a methodology which matches training needs with training emphasis in 
order to link training evaluation information to training needs reassessment 
and program revision. Two illustrative examples of this matching process — 
one involving a police recruit training program and another involving a 
supervisory skills training workshop are presented. 

SECTION I 

LINKING JOB ELEMENTS WITH A 
CONTENT VALIDITY RATIO APPROACH 
TO ESTABLISH JOB RELATEDNESS 

In the first section, a methodology for establishing the job relatedness 
of a police recruit training program is presented. As background for this 
section, a description of the police training program is given and the design 
issues faced by the researchers are described. 
Police Training Program 

The objective of the recruit training program is to prepare entry level 
officers to perform the police patrol function. Therefore, the curriculum has 
been developed to instill the knowledges, skills and attitudes necessary for 
efficient and equitable patrol operations. To best meet its stated objective, 
the recruit training program has been expanded over the last few years from a 
three month program to a six month, 800 hour program. Courses have been added 



or dropped and course lengths increased or decreased based primarily on the 
needs perceived by training staff members or administrative officials. This 
project qualifies as the first systematic evaluation o£ the training program 
aince its expansion* 
Design Issues 

After a thorough review of the relevant literature it was decided that the 
goals of the project to establish job relatedness and determine training needs 
could best be met by linking a content validity strategy with a modified job 
element approach to- job analysis. 

Job Element Approach . The job element approach requires the specification 
of the knowledges, skills, abilities and other personal characteristics (KSAO's) 
that are necessary for job performance. This approach is normally applied to 
determine the content validity of tests by directly matching test items to the 
job elements. Since the police recruit training academy attempts to instill or 
impart certain knowledges, skills and attitudes to the recruits, a job element 
approach seems especially applicable in determining job relatedness. In 
addition, the comprehensive analysis of the KSAO's taught in the training 
academy provided the level of specificity needed to determine training needs 
and initiate program revisions. 

Content Validity as a Strategy . The notion of content validity as one 
aspect of the more general notion of validity, refers to an evaluation of the 
representativeness or adequacy of sampling the content domain of the job (Ebel, 
1977; Tenopyr, 1977). As a strategy to determine job relatedness, it has 
generated theoretical debates (Guion, 1978; 1980; Tenopyr, 1977) and legal 
case support (Kleinman and Faley, 1977). It is not our purpose to review this 
controversy. Instead, it is noted that the Division 14 Principles for Validation 
(American Psychological Association, 1980) and the Uniform Guidelines (Equal 
Employment Opportunity Commission, 1978) state that selection procedures which 
purport to measure knowledges, skillF or abilities (KSAO's) may be justified 



by content validity if the procedure can be operationally defined and if ihc 
KSAO's are prerequisites to successful job performance. The selection proce- 
dure in our case is a police training program which purports to train certain 
KSAO's which police officials consider prerequisites to successful job perfor- 
mance. Therefore, a content validity strategy was seen as well suited for the 
purpose of determining the job relatedness of the training program. 

A content validity strategy for a training program consists of demonstrating 
the extent of relationship between the content of the program and the content 
of the job. The first step in this process is to identify the content. A 
second step is to evaluate the content. For this project, the content was 
identified through a job element approach while a content validity ratio 
approach (Lawshe, 1975) was used to evaluate the content. 

PROJECT DESIGN 

An overview of the project design is presented in Figure 1. To describe 
how the goalsfor this project were' addressed, the project is divided into two 
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phases. The first phase consisted of the identification of the training and 
job content domains. For the second phase, the job relatedness of the training 
content domain was evaluated through a content validity ratio approach. 

Phase One: Identification of Content Domains 

Identification of the Training Content Domain 

The first and most important aspect of this project was to accurately 
identify the content domain of the training program. The content domain was 
defined in terms of the knowledges, skills, abilities and other personal 
characteristics (KSAO's) currently taught or developed in the training prog 



ram. 



insure comprehensiveness, the KSAO's were derived from a three step process 
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structured around a series of panel sessions and a large scale survey (see 
Table 1). 
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Step 1: Generation of KSAO Statements > In the first step, three indcpcin- 
dent panels were created and held responsible for generating a list of KSAO's 
that was an accurate and comprehensive reflection of training content. Two of 
the panels consisted of Training Academy staff members and instructors and the 
third panel consisted of members of the most recently graduated recruit class* 
Each panel met for 3 four-hour sessions or a total of 12 hours. 

Panel sessions were conducted according to a modified brainstorming ap- 
proach (e.g., see Primoff, 1975). Members were asked to generate lists of 
KSAO's that were currently taught or developed in the training program. The 
process was loosely structured in which panel members suggested relevant 
statements or topic areas. Panel members then discussed and refined the 
suggested statements before they were recorded. Midway through the sessions, 
the training program syllabus and training materials were introduced to stimu- 
late discussion and insure comprehensiveness. A total of 963 KSAO statements 
were generated by this process across the three panels. 
Step 2: Revision of KSAO Statements 

For the second step, the three lists of KSAO's were combined, the state- 
ments were categorized into subgroups and duplicates were eliminated. A 
KSAO Revision Panel of training staff members was then formed to refine the 
list of categories and to edit, reword and combine the existing statements to 
insure comprehensiveness and reduce ambiguities. The final list of KSAO's, as 
revised by the Revision Panel, and the research staff, consisted of 393 KSAO 
statements. 
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' Step 3; Training Content Doroaln Survey 

For the third step, a survey was developed to insure that the training 
content domain had been accurately and comprehensively identified,. The 

survey, called the Training Content Domain Survey, consisted of the 393 KSAO 

statements identified through the panel sessions. Participants were asked to 

respond to each KSAO statement in terms of whether the KSAO was taught or 

developed in the training program. Analysis of the "taught" scale helped to 

solidify the identification of the training content domain. Participants who 

completed the survey included 15 training staff members and 35 police officers 

from the most recent recruit class. 

As a result of the survey, 10 KSAO statements were eliminated as not 

actually being taught or developed in the training program. Consequently, the 

content domain of the training program was defined as consisting of 383 KSAO's. 

Job Content Domain 

Another component in the first phase of the project was to define the 
content domain of the patrol officer's job in terms of the knowledges, skills, 

abilities and other personal characteristics (KSAO's) needed to perform. This 

information could then be used to identify areas in which the training program 

was deficient or in which training content was overemphasized. The job domain 

was also specified through a three step process (see Table 2) . 



Present Table 2 about here 

Step 1; Initial KSAO List 

To develop a comprehensive list of KSAO's needed to perform on the job, 
two independent sources of KSAO's were available. The first source was the 
indepth analysis of the training content domain previously discussed. Since 
the training program is geared to training recruits to be patrol officers, the 
383 KSAO statements provided a logical and reliable source of information. A 
Q cond independent list of KSAO statements which could define the patrol offi- 
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ccr*s job was then drawn from job analyses from other police jurisdicLionb 
which utilized a KSAO approach. After the elimination of duplicates, a total 
of 373 KSAO statements reniained from this second source of statements. 
Step 2; Combination and Revision of KSAO Statements 

For the second step, two panels of 11 officers and sergeants were formed. 
Each panel met on three separate occasions for a total of 10 hours for each 
panel* The major objectives of the session were to review, edit and revise 
the 373 KSAO statements derived from other police job analyses to insure their 
relevancy to the patrol officer's job in the target city« As a result of the 
panel sessions, 80 KSAO statements were eliminated and 38 were added. Conse- 
quently, 331 KSAO's were identified as possibly defining the job of patrol 
officer in this midwest ern city. 

At this point the research staff had two independent lists of KSAO state- 
ments which could define the content domain of the patrol officer's job; 
i.e., the 383 KSAO's from the training content domain and the 331 KSAO's from 
other police job analyses. For the purposes of the project, it was necessary 
to develop one comprehensive list which defined the job content domain. 

To develop one comprehensive list to define the job content domain, inde- 
pendent panels of officers placed the two lists of KSAO's into job relevant 
major categories and a number of subcategories (see Table 3). Since there 
was a large degree of overlap between the two lists of KSAO's, it was decided 
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to use the training content domain as a starting point for defining the job 
content domain. The research staff independently rated whether a KSAO state- 
ment from the other police job analyses was similar or identical in content to 
one of the 383 KSAO's that defined the training content domain. Using these 
ratings to help come to a group consensus, those considered a duplicate were 
O liminated from further consideration. Through this review process, 57 KSAO 



7 

ft 

atatements were identified as being unique in content from the existing train- 
ing content domain. As a result, the job content domain of patrol officers 
was tentatively defined as containing the 57 unique KSAO's and the 383 training 
content KSAO's for a total of 4A0 KSAO statements. 
Step 3: Job Content Domain Survey 

For the third step, the 440 KSAO statements which tentatively defined the 
job domain were used to form a KSAO inventory called the Job Content Domain 
Survey. The Survey was administered to 296 patrol officers and 31 patrol 
sergeants from the target city. Patrol sergeants and officers were asked to 
read each KSAO statement and to decide if the KSAO was required to perform the 
patrol officer's job. This information was then utilized to finalize the 
identification of the job content domain. Results of the analyses indicated 
that three items that were part of the 57 unique KSAO's were not relevant to 
the job. Therefore, the job content domain was defined as consisting of 437 
KSAO statements* 

Phase Two: Evaluation of Job Relatedness 

In phase one, it was found that the 383 KSAO's which defined the training 
content domain were also required to perform the job of patrol officer. In 
addition, 54 KSAO statements were identified which were required for job per- 
formance but not incorporated into the training program. The fact that only 
54 of the 437 KSAO's which defined the job content domain were unique, provides 
descriptive support for the job relatedness of the training content domain. A 
more systematic assessment of the job relatedness of the training content 
domain was established through a content validity ratio approach (Lawshe, 1975) . 
Content Validity Ratio Approach 

The content validity ratio approach was initially introduced by Lawshe to 
aid the development of content valid test items. The approach, though, appears 
to be especially well suited for evaluating the job relatedness of the train- 
content domain. 
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The technique requires the use of "subject matter experts" - people who 
have a thorough knowledge of the job - who rate the importance or essentiality 
of each item (KSAO) for job performance. Two assumptions of this approach are 
that any item which is perceived by more than 50Z of the experts to be important 
has some degree of job relatedness and that as more experts beyond 50Z perceive 
the item as important, the greater the extent or degree of content validity. 
Table A presents the formula for calculating a content validity ratio (CVR) 
and some defining characteristics of the ratio • 
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The ratio is calculated by simply taking the number of experts who state 
that an item is important, subtracting from that number the number of experts 
who state the item is not important, and dividing by the total number of experts. 
As you can see from the table, a CVR will be negative when fewer than half the 
experts say an item is important, positive when more than half say an item is 
important and zero when exactly half the experts believe an item is important 
to job performance. As a final note, an overall summary of the content 
validity of the training program can be generated by taking the mean of all the 
CVR values. This summary statistic also presented in Table 4 is called the 
Content Validity Index (CVI). A statistical test of significance for CVR 
values has also been developed by Lawshe (1975) and Schipper (Note 1). The 
statistic is based on the probabilistic notions in that CVR values are examined 
to determine if they depart significantly from a CVR value of 0 (i.e., where 
half the experts agree the item is important while half agree the item is 
unimportant). Table 5 presents the minimum CVR values necessary for a one- 
tailed test (p < .05) for various sample sizes. As you would expect, as sample 
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sire increases the minimuiD CVR values needed for the traditional declnratiDn 
of significance decreases. For example, with 20 experts a CVR value above 
.42 is considered significant while with a sample size of 60 a CVK value of 
•23 Is required. 
Application of CVR Approach 

For the present project, three sources of "subject matter experts" were 
used for a total sample size of UA. The experts included 64 experienced 
patrol officers who trained recruits on the job (Field Training Officers), 
31 patrol sergeants and 20 police personnel from similar sized communities out- 
side the target city. The experts independently rated the importance of each 
KSAO in the training content domain for job performance (the 54 additional KSAO'S 
in the job content domain were also rated for importance by the sergeants and 
the police personnel from the other communities). The importance ratings were 
made on a seven point scale which ranged from "no importance" to "extreme 
importance". A decision rule was enacted whereby KSAO's with ratings of five, 
six or seven were considered important or essential while KSAO*s with ratings 
less than five were considered relatively unimportant. The degree of consensus 
among the experts was than quantified by calculating a content validity ratio 
for each KSAO statement. 
Results 

Figure 2 presents a frequency distribution which represents the number of 
KSAO statements for various CVR values. As can be seen from the figure, there 
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was a large range of CVR values from -.85 to +.93 with a modal response within 
the +.47 to +.57 interval. With a sample size of 114, the CVR significance 
level is approximately .17. Of the 383 KSAO's defining the training content 
domain, 237 or 62Z of the KSAO*s were above that significance level. 
Q Table 6 presents the mean CVR value of the 383 KSAO's, i.e., the content 
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validity index (CVI) for the target city sample and the outside city sample. 
As can be seen from the table, the CVI*s for both samples were significant. For 
the total combined sample of 114 experts, the content validity index was .27 
(p < •05). The results demonstrate that the content of the training program 
has a significant degree of job relatedness. 

Whenever experts make judgements, a crucial question is the quality of 
their judgements. If the three sources of experts do not agree regarding the 
Importance of the KSAO's, serious questions about the usefulness of the results 
can be raised. When they do agree, we must conclude that the experts are either 
"all wrong" or "all right" (Lawshe, 1975). Because the experts in this case 
are performing the job in question or are in direct supervision, there is 
little basis for refuting strong agreement. Table 7 presents the intergroup 
correlations for the officer, sergeant and outside city police rating groups. 
The results indicate that patrol sergeants and patrol officers as well as 
police personnel from outside cities have very similar perceptions of how 
important the various job elements are to job performance. This high level 



Present Table 7 about here 
of intergroup agreement (r > .80) supports the quality of the data collected. 
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SECTION II 

LINKING TRAINING EVALUATION TO TRAINING 
NEEDS REASSESSMENT AND PROGRAM REDESIGN 

As often happens in organizations, pressures to develop and quickly imple- 
ment a training program to address perceived deficiences prevent a thorough 
analysis of training needs. In addition, once a program is established, train- 
ing needs may change over time. Such a situation indicates a critical need to 
not only evaluate the effectiveness of an existing training program, but also to 
link that evaluation to a reassessment of training needs and a redesign of the 
training program. A model of this evolutionary process is presented in Figure 3. 
Unfortunately, little attention has been given to methodologies which establish 
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this critical linkage, allowing the training process to evolve from Phase IV, 
training evaluation, to Phases V 5 VI, reassessment and redesign. This section 
describes one method for establishing this linkage - the matching technique of 
matching training needs with training emphasis. 

The matching technique is conceptualized in Figure 4 as a matrix in which 
training emphasis is directly compared to training needs. Such a comparison 
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is a powerful method as it allows for the identification of training "hits" and 
"misses." The hits refer to those content areas, such as KSAO's, where the 
emphasis received in training is confirmed or justified. Areas that need to 
be trained are trained; areas that do not need to be trained are not trained. 



The training misses can be separated into two types: deficiencies and 
excesses. Of the two, deficiencies would generally be a more critical concern. 
Training deficiencies are content areas whose high training needs suggest more 
emphasis is required than is currently received in training. The extreme case 
would be a training omission, where a training need is not emphasized at all 
in training. On the other hand, training excesses are areas receiving an ex- 
cessive amount of emphasis, relative to their need to be trained. 

The greater the number of misses uncovered, both deficiencies and excesses, 
the less effective the program and, therefore, the greater need for refine- 
ment or redesign. This matching technique provides information as to what 
specifically in the training program should be changed and in what way. As an 
example, training needs and training emphasis will be matched first within the 
context of the Police recruit training program. To demonstrate the versatility of 
the technique, it will also be further applied within the context of a super- 
visory skills workshop in an industrial setting. 
Application to a Police Training Program 

The job relatedness of the training program has been established through 
a content validity ratio approach. Equally important the range of CVR values 
for the training and job content domains provides an empirical base to system- 
atically '^-easse^s training needs for program revision. 

During the evaluation phase of our project, the KSAO statements were 
separated into the seven major categories and the number of minor categories 
presented earlier (see Table 3). These categories reflect the core curriculum 
taught in the training program. The appropriate KSAO CVR's were then averaged 
to calculate a CVI for each category and subcategory. 

Initially, these CVR and CVI values were used to determine job relatedness. 
For the matching technique, since they indicate importance to job performance, 
these same CVR and CVI values can be used to determine training needs. 
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Training emphasis consisted of the class time devoted to each category and 
subcategory in the training program. Finally, both the needs and emphasis 
scales were divided into quartiles, based on the range of CVI values and 
class time, respectively. 

As the first step in the matching process, Figure 5 matches the CVI's 
for the seven major categories and the class time devoted to each category. 
The results indicate major support for the time spent on the various content 
areas. Only two categories ~ Laws (1) and Divisional Directives (3) — 
deviate even slightly from a perfect linear relationship. Since training 
deficiencies are the more critical concern, the sole deficient category of 
laws requires further investigation. 
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Figure 6 illustrates the same matching technique but at a more detailed 
level of specificity to discover why the Law Category was slightly training 
deficient. Training time and the CVI values are matched for the six law 
subcategories. Results indicate a close match, but less so than in Figure 5. 
A major training excess occurs in the content area of Criminal Laws, suggesting 
less emphasis be placed there. Conversely, there are three subcategories that 
are slightly deficient, which in combination resulted in the Law Category's 
slight overall deficiency. Among the three areas, more deficiencies occurred 
in Rules of Evidence, suggesting more emphasis be placed in teaching Rules 
of Evidence. 
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Table 8 presents information about what KSAO's should be the focus of the 
increased emphasis on Rules of Evidence. The CVR values are listed for the nine 
KSAO's grouped under Rules of Evidence. Higher CVR values suggest greater 
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emphasis. 
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This information is not only useful for allocating time within courses but 
also provides information to evaluate and revise testing procedures. For 
example, the rules of evidence exam given recruits can be examined to insure 
that the knowledge items with high CVR values (e.g., Knowledge of what is or 
is not evidence) are emphasized on the test, while knowledge items with low 
CVR's (e.g.. Knowledge of privileged communications) are given relatively 
little emphasis on the test. 

In addition to ratings of training needs (CVR), each KSAO was rated by 
the police training staff and recent recruits in terms of training emphasis. 
This information allowed for a direct comparison of emphasis with training needs 
which resulted in a scatterplot of the 383 training content KSAO's. The corre- 
lation of the emphasis and needs ratings were quite high (r .68, p < .001) 
indicating a large proportion of "hits," or training confirmations, with few 
KSAO's seen as being highly deficient or excessive. Nevertheless, as discussed 
previously, the job content domain had been identified as consisting of 54 
KS.AO's which were not included in the training program. These training omissions 
were examined in terms of their trainability within the constraints of the re- 
cruit training program. The trainable KSAO*s have recently been incorporated 
into the training program while other KSAO's have become emphasized to a greater 
extent in the field training recruits receive once completing the academy 
training program. 

Application to a Supervisory Skills Workshop 

The matching of training needs to emphasis can also be successfully applied 
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to an industrial setting* The training program to be reassessed and rede- 
signed is a supervisory skills workshop at Ethyl Corporation called "TIPS'* - 
Training in People Skills. Unlike the police academy, which is a compre- 
hensive job training program, the content domain of the TIPS workshops is a 
subset of a supervisor's required job skills* specifically, a subset of 15 
interpersonal situation skills (see Table 9). The method for training these 
skills is a "participatory" applied learning approach (Wroten, Note 2). 
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Initially, TIPS was designed from an informal assessment of needs, based 
on talks with plant managers and the supervisors themselves. However, as in 
the police study, there was the opportunity to incorporate into the training 
evaluation process the means to reassess training needs for program redesign 
purposes. Thus, training needs were more systematically determined by getting 
subordinate, managerial and self ratings. They rated the perceived effective- 
ness of the supervisor on each of the 15 interpersonal situation skills. Based 
on the average for all untrained supervisors, the skills were then divided 
into the top, middle and bottom third in terms of needing training. 

The training emphasis given to each situation skill was determined from training 
evaluation results (Wroten, Note 2). In an analysis of covariance, the difference 
between trained and untrained supervisors was deteimined on each situation skill. 
A bigger difference (i.e., F-value) implies a bigger emphasis or impact of the 
training in that situation skill. Thus, the 15 situation skills could then be 
divided into the top, middle and bottom third in terms of emphasis received in 
training. 
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Figure 7 presents the matrix matching the training needs of the untrained 
supervisors and the training emphasis of the TIPS workshops. The matrix in- 
dicates that the overall fit of the training program to the average supervi- 
sor's needs is relatively good, with few extreme training deficiencies or 
excesses. 
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Nevertheless, an advantage of the matching technique is tc fuither refine 
a training program by identifying relative extremes. Thus, the emphasis on 
skill 10 - "handling a complaint or grievance from an employee** - (and to a 
lesser extent skills 4,6 and 14) could be reduced and other skill areas 
(1,3,7,9 and 12) could be emphasized more. 

Another key advantage of the matching technique lies in its potential 
for customizing training programs to better meet the needs of organizational 
subunits, such as individual plant locations. Once the match is determined for 
each plant, the training program can be refined and conducted at the plant to 
maximize effectiveness. To illustrate this advantage. Figure 8 presents the 
matrix for supervisors at one Ethyl plant. The results indicate that the training 
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emphasis in the current program is not in tune with the training needs of the 
supervisors at that particular plant. Since the skills workshop was specifi- 
cally structured in terms of training modules, it is flexible enough to make 
adjustments in content based on such analyses. Therefore, in this plant, the 
training program could be customized to increase the emphasis given to seven skills 
(1,3,8,9,11,12 and 15) while five skills (5,6,7,10 and 14) could be deemphasized 
or perhaps eliminated from the program. 
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Decisions on how to further refine the program such as which of the 
many deficiencies should receive the most emphasis ~- can be aided by a 
dollar impact analysis ^that supplements the matching technique. The dollar 
impact analysis consisted of measuring the dollar effect of the overall changes 
in the supervisor, as estimated by the supervisor's raters. This was corre- 
lated with how much that supervisor changed on each situation skill, as ob- 
served by the raters. Changes in seven of the 15 situation skills are signi- 
ficantly correlated (p < ^10, N = 66 supervisors) with the overall dollar 
effect of those changes. They are skills 1,3,4,5,8,9 and 10. 

These results indicate which changes in situation skills would have a 
greater dollar impact and should therefore receive greater attention. Applying 
this to the results of the matching technique, three of the five training ex- 
cesses are not related to dollar impact, suggesting emphasis be shifted away from 
skills: 

6. Conducting a follow-up discussion with an employee 
about poor performance or work habits. 

7. Motivating and encouraging a poor or average per- 
former; recognizing average, but not outstanding work. 

14. Dealing with peers. 
Instead, the emphasis should be shifted especially to the four, out of the 
seven, training deficiencies that are related to dollar impact: 

8. Setting performance goals or standards and reviewing 
performance with an employee. 

3. Delegating tasks, responsibilities and authority. 
1. Coaching and giving on-the-job training. 

9. Handling disciplinary matters. 

ERIC Id 
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By basing such redesigns of a training program on such reassessments 
of current training needs, the training program will be better fit to 
address the organization's needs, thus assuring the training program^s sur- 
vival and its continuing to further evolve. 



Conclusion 

As stated at the beginning of this presentation, very little research 
has been conducted which offers methods which (1) establish the content 
validity of a training program, and (2) translate training evaluation infor- 
mation into training needs for program revision. We have offered a methodology 
of linking a job element method of job analysis with a content validity ratio 
approach to establish job relatedness. In addition, we have offered a 
methodology of matching training needs with training emphasis to link training 
evaluation with training needs reassessment. 

To quote Cascio (1978), "if you don't know where you are going, any road 
will get you there, but if you do know where you are going, you increase your 
chances of actually getting there" (p. 282). While our approaches certainly 
have some weaknesses, it is our hope that we have presented methodologies which 
offer new roads to travel and spur further efforts to opening more roads in the 
training literature so as to increase our chances of actually getting somewhere. 
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Table 1 



TRAINING CONTENT DOMAIN - Identify Che knowledges, skills, 
abilities and other personal characteristics taught 
or developed. 



STEP 1: Generation of knowledges, r^kills and abilities 

- Panel sessions with Training staff, instructors 
and recent recruits • 



STEP 2: Revision of KSAO Statements 

- Panel Sessions and Research Staff 



STEP 3: Completion of Training Academy ContonL Survey 



A, Survey Participants: 

- Training Staff 

- Recruit Class 



n= 15 
n= 35 



B, Information From Survey 
- Taught or not 



Tabic 2 



JOB CONTENT DOMAIN - Identify the knowledges, skills, abilities 
and other personal characteristics necessary to perform the 
patrol officer's job. 

STEP 1: Initial KSAO List 

A. Training Content. Domain 

B. Other Police Job Analyses 

STEP 2: Combinat ion and Revision of KSAO Statements 
^ A. Panel Sessions with Police Personnel 

B. Combination of lists by Research Staff 

STEP 3; Completion of I^aLrol Officer Knowledges and Ability 
Survey 

A. Survey Participants 

- Patrol Officers 

- Patrol Sergeants 



B. Information from Survey 

- Required for Job Performance 



Tabic 3 

Training Academy Content Domain: Categories For KSAO Statements 

1 * Laws 

A. Criminal Laws 

B. Traffic Laws 

C. Arrest J Search, and Seizure 

D. Rules of Evidence 

E. Liability 

Procedural Law (warrant, subpoenas, summons, complaints) 

2. Investigation and Observation 

A. Traffic and Accident 

B. General Crime and Crime Scene 

C. Observation and Surveillance 

3. Divisional Directives 

A. Policies and Procedures (non-mandated by law) 

B. Military Conduct 

C* Division Organization and Operations 

4. Courts 

A, Court Structure 

B, Court Appearance and Toscijnony 

C, Ccurt Room Procedures 

5. Physical Characteristics and Situations 

A. Physical Attributes and Fitness 

B. Firearms 

C. Intermediate Weapons 

D. Special Tactics 

E. Arrestee Control and Use of Force 

F. Officer Safety and Survival 
C. Equipment Use and Operation 

6 • Commun ic a t ion 

A. Verbal/Oral Communication 

B. Non-Verbal Communication 

C. Written Communication 

7. Human and Community Relations 

A. Crisis Intervention 

B. Media Relations 

C. Community Services and Other KcJations 

D. Assessment & Management of Persons & Situations 

E. Personal and Self-Manai;enient 



Table 4 

Quantification of Content Validity 



A. CONTENT VALIDITY RATIO KOKMULA 



CVR -= 




u 



Where: number of 



N » Experts rating KSAO as important 



N^ *= Number of experts rating KSAO 



as unimportant 



N,p = Total number of experts 



B. CHARACTERISTICS 

1 . CVR is NEGATIVE when fewer than half say an item is 
"important" 

2. CVR is ZERO when half say item is "important" 

3. CVR is POSITIVE when more than half say item is 
important 

A. CVR is ONE when all experts say item is important 



C. CONTENT VALIDITY INDEX 



CVI «= (CVR) 



^From Lawshe (1975) 



Table 5 



Minimum Value of CVR^ 



One 


Tailed Test P - . 




Minimum 


N 


Value 


5 


.99 


7 


.97 


9 


.78 


10 


.62 


15 


.49 


20 


.A2 


25 


.37 


30 


.33 


35 


.31 


AO 


.29 


50 


.252 


60 


.229 


70 


.211 


ou 




90 


.185 


100 


.175 


*110 


.166 


120 


.158 


130 


.152 


lAO 


.1A6 


150 


.141 


From 


Schipper (1980) 
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Table 6 
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Overall CVI's By Subgroup 



Overall CVI Minimum CVI 

Sample (383 KSAO's) For si^^nif icance 

Target City .22 .19 

Sample 
(N«99) 

Outside City .49 .42 

Sample 
(N=20) 



Total Combined .27 .17 

Sample 
(N=11A) 



4. / 



Table 7 



Correlations An)onfi Subgroun Samples 
On the CVR Ratings 



Outside 

Target City Td.gct City City 

Officers Sergeants Personnel 

(N° 64) (N'= 30) (N^ 20) 



Target City Officers 

Target City Sergeants ,85 

Outside City Personnel .80 .80 



a 

All correlations are significant, p > .001, N= 383 KSAO*s, 
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Table 8 
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CVR Values for KSAO's 
in Rules of Evidence Subcategory 



KSAO 



* p < .05; N = 114 



2j 



CVR 



• Knowledge of rules of evidence* (-54)* 

• Knowledge of how to maintain chain-of-custody for evidence. (-58)* 

• Knowledge of the legal rationale behind various rules of (--10)* 
evidence. 

i Knowledge of "corpus delicta" and what evidence should be (-42)* 

presented in association with the crime committed. 

• Knowledge of different types of forms of evidence (such as, (-19)* 
real, circumstantial and direct evidence; declarations and 
confessions) . 

• Knowledge of privileged communications. (-•07) 

• Knowledge of what is or is not evidence, and what constitutes (.78)* 
its admissibility, relevancy and competency. 

• Knowledge of who shall, when to, and how to legally obtain (-66)* 
evidence at the scene. 

• Knowledge that a judge rules on relevancy and competency of (•IS) 
evidence . 



Table 9 

EthyPs •'TIPS" Workshops (Training in People Skills) : 
15 Supervisory Interpersonal Situation Skills 

1. Coaching and giving on-the-job training. 

2. Correcting a situation immediately and then explaining to % 
the employee what the problem was. 

3. Delegating tasks, responsibilities and authority. 

4. Dealing with an employee's performance problem (such as a 
drop in quality) . 

5. Dealing with poor work habits (such as excessive lateness). 

6. Conducting a follow-up discussion with an employee about 
poor performance or work habits. 

7. Motivating and encouraging a poor or average performer; 
recognizing average, but not outstanding, work. 

8. Setting performance goals or standards and reviewing 
performance with an employee. 

9. Handling disciplinary matters. 

10. Handling a complaint or grievance from an employee. 

11. Overcoming an employee's resistance or resentment to changes 
or unpopular decisions. 

12. Counseling on personal or career matters. 

13. Chairing and conducting meetings or participating in meetings 
or project groups. 

14. Dealing with peers. 

15. Dealing with superiors. 
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Figure 1 
Project Design 



Coal: Job Rclatedncss of 
Training Program 
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Figure 2 

Frequency of KSAO CVR's for Total Sample (N = 11 A). 




Figure 3 

THE EVOLUTION OF THE TRAINING PROCESS 




Initial 
Needs 
Assessment 



IV 

Training 
Evaluation 



II 

Program 
Development 



ERIC 



III 

Program 
Implementation 
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Figure 4 



Matching Technique: 

Matching Training Needs to 
Training Emphasis 



Low Training Emphasis High 



Training 
Needs 




3j 
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Figure 5 



Matching Technique for 7 KSAO Categories' 



TRAINING EMPHASIS 
(Hours) 



LOW 

0 



HIGH .5 



TRAINING 
NEED 



(CVI's) 



.3 



LOW 0 



50 



100 



150 



HIGH 
200 



5,7 



^ The numbers in the matrix indicate the 7 major KSAO categories 
(see Table 3) . 



Figure 6 



Matching Technique for 6 Law Subcategories' 



LOW 
0 



HIGH .8 



TRAINING 
NEED 

(CVI's) 



.4 



LOW .0 



TRAINING EMPHASIS 
(Hours) 



12 25 



38 



HIGH 
SO 









C 


D 








E.F 


B 




A 











^ The letters in the matrix indicate the 6 subcategories of the Law 
Category (see Table 3). 
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Figure 7 



Matching Technique for 
15 Interpersonal Situation Skills 





Low 


Training limphasis 


High 


High 












Q,y,n 


©.13 


Training 
l))eeds 




2 


6,1A 


Low 






@ 



The numbers in the matrix represent the 15 interpersonal situation 
skill areas. For circled numbers 1,3,4,5,8,9 and 10, changes 
in those situation skills are significantly correlated (9 < .10) 
with the overall dollar impact of training. 
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Figure 8 



Matching Technique for 15 Interpersonal Situation Skills at 

One Plant Location 





Low 


Training Emphasis 


High 


High 












a).i2 


13 


Training 
Needs 




2, A 


14 






(D.7 




Low 









The numbers in the matrix represent the 15 interpersonal skill 
areas. For circled numbers 1,3,4,5,8,9 and 10, changes in those 
situation skills are significantly correlated (p < .10) with the 
overall dollar impact of training. 
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